Systems and methods for navigating through airways in a virtual bronchoscopy view

ABSTRACT

Systems and methods for displaying virtual bronchoscopy views while navigating through an airway of a virtual bronchoscopy are disclosed. The method comprises determining a first location and a first direction at the first location, storing the first location and the first direction in memory, displaying a first virtual camera view corresponding to the first location, determining a second location corresponding to movement through the airway of the virtual bronchoscopy, storing the second location in the memory, displaying a second virtual camera view corresponding to the second location, determining a second direction based on the first location and the second location, storing the second direction in the memory, determining a third location corresponding to further movement through the virtual bronchoscopy, and determining whether the further movement is in a forward direction or a backward direction.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 15/184,057, filed on Jun. 16, 2016, which claimsthe benefit of and priority to U.S. Provisional Patent Application No.62/181,824, filed on Jun. 19, 2015, the entire contents of each of whichare incorporated herein by reference.

BACKGROUND Technical Field

The present disclosure relates to systems and methods for displayingmedical images in a dynamic and changing manner. More particularly, thepresent disclosure relates to systems and methods for dynamicallydisplaying medical images during forward and backward movement in avirtual bronchoscopy during pathway planning based on a position and adirection of a virtual camera being navigated through airways of thelungs.

Discussion of Related Art

Pathway planning visualization techniques are rapidly growing in variousmedical areas. Visualization techniques help minimize the size of anincision, non-invasively treat diseases of a patient in surgeries, andnon-invasively navigate inside of patients to identify and treat targetlesions. However, visualization may pose unexpected risks when incorrectinformation is displayed. For example, when navigating a virtual cameraview in a backward or retracting direction through an airway of thelungs, the views provided to a clinician may make it difficult for theclinician to retrace the previously taken steps and return to a previouslocation. Moreover, a user may find the virtual camera view navigatingoutside of the airways of the lungs.

SUMMARY

In one aspect, the present disclosure features a method for displayingvirtual bronchoscopy views while navigating through an airway of avirtual bronchoscopy. The method includes determining a first locationand a first direction at the first location, storing the first locationand the first direction in memory, displaying a first virtual cameraview corresponding to the first location, determining a second locationcorresponding to movement through the airway of the virtualbronchoscopy, storing the second location in the memory, displaying asecond virtual camera view corresponding to the second location,determining a second direction based on the first location and thesecond location, storing the second direction in the memory, determininga third location corresponding to further movement through the virtualbronchoscopy, and determining whether the further movement is in aforward direction or a backward direction.

If it is determined that the further movement is in the forwarddirection, the method includes displaying a third virtual camera viewcorresponding to the third location. And If it is determined thatmovement is in the backward direction, the method includes retrievingthe stored first direction and the stored second direction from thememory, determining a smoothing vector based on the stored firstdirection and the stored second direction, obtaining a smoothed virtualcamera view based on the smoothing vector, and displaying the smoothedvirtual camera view.

In embodiments, determining whether the further movement is in a forwarddirection or a backward direction includes determining a next pointerlocation and a current pointer location on a virtual bronchoscopyscreen, calculating the difference between a coordinate of the nextpointer location and a coordinate of the next pointer location. If thecalculated difference is positive, determining that the further movementis in the backward direction. And if the calculated difference isnegative, determining that the further movement is in the forwarddirection.

In embodiments, the first location, the second location, and the thirdlocation are determined based on the location of a pointer or cursor ona screen which is displaying the virtual camera views. In yet anotherembodiment, the method further includes that if it is determined thatthe further movement is in the forward direction determining a thirddirection based on the second location and the third location, andstoring the third direction in the memory.

In embodiments, determining the smoothing vector includes determining afirst vector based on the first location and first direction,determining a second vector based on the first location and the seconddirection, and averaging the first vector and the second vector toobtain the smoothing vector.

In embodiments, the spline is a spline of order two or a spline of orderfour. In embodiments, the method further includes receiving an inputfrom a user which alters the first, second, third, or smoothed virtualcamera views, and storing the altered first, second, third, or smoothedvirtual camera view. In a further embodiment, determining the smoothingvector includes determining a vector having a direction between thefirst direction and the second direction.

In another aspect, the present disclosure features an apparatus fordisplaying virtual bronchoscopy views while navigating through an airwayof a virtual bronchoscopy. The apparatus includes a network interfaceconfigured to receive position information of a navigation instrumentfrom at least one position sensor of the navigation instrument, theposition information including physical locations, a memory storing aplurality of virtual camera views of the virtual bronchoscopy,instructions, a first location, a first direction at the first location,a second location, and a second direction at the second location, aprocessor configured to execute the instructions. The instructions, whenexecuted by the processor, cause the processor to determine whethermovement through the airway of the virtual bronchoscopy is in a forwarddirection or a backward direction. If it is determined that the movementis in the forward direction, the instructions further cause theprocessor to determine a third location corresponding to the movementthrough the airway of the virtual bronchoscopy, and determine a thirddirection based on the second location and the third location. If it isdetermined that movement is in the backward direction, the instructionsfurther cause the processor to retrieve the first direction and thesecond direction from the memory, and determine a smoothing vector basedon the first direction and the second direction. The apparatus furtherincludes a display configured to dynamically display, on a screen,images of a smoothed virtual camera view corresponding to the determinedsmoothing vector.

Any of the above aspects and embodiments of the present disclosure maybe combined without departing from the scope of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Objects and features of the presently disclosed systems and methods willbecome apparent to those of ordinary skill in the art when descriptionsof various embodiments are read with reference to the accompanyingdrawings, of which:

FIG. 1 is a schematic diagram of a computing device for pathway planningin accordance with an embodiment of the present disclosure;

FIG. 2 is a diagram illustrating the four phases of pathway planning inaccordance with the present disclosure;

FIG. 3 is a flowchart illustrating a method for dynamically determiningand displaying virtual camera views for forward and backward motionwithin lung airways according to an embodiment of the presentdisclosure;

FIG. 4 is a flowchart illustrating a method of determining forward orbackward direction within lung airways according to an embodiment of thepresent disclosure;

FIGS. 5 and 6A-6C are graphical illustrations of views while navigatingwithin lung airways in accordance with an embodiment of the presentdisclosure; and

FIG. 7 is a graphical illustration of steps taken to navigate throughlung airways in accordance with some embodiments of the presentdisclosure.

DETAILED DESCRIPTION

The Virtual Bronchoscopy (VB) view enables a user to interact with avirtual camera (or point of view of the user) and to modify both thelocation and direction of the virtual camera view inside the airways oflungs. The movement backwards in the VB view could be achieved byadjusting the virtual camera (or user point of view) on one of theplanar views (Axial, Coronal, Sagittal) and then going backwards in astraight line in the VB view. However, this method may involve at leasttwo views and it may be difficult to retrace the actual steps and returnto a previous location. Also, a user may find the virtual cameranavigating outside the airways.

According to embodiments of the present disclosure, the movementforward, including turns, is recorded by storing each individual step ina stack and when the virtual camera moves backwards, its backward stepis taken from the stack, therefore retaining both the actual locationand direction of the virtual camera. Movement forward is performed bymoving in straight lines. Movement sideways is performed by operating adata input device such as a mouse pointer at a position on the VB viewand the center of the VB view updates to that position by way ofanimation to allow for a better user experience.

When the virtual camera view is turned (to the right or to the left)while moving forward in response to movement of a cursor or pointer onthe screen and/or selection of a button by a user operating a user inputdevice such as a mouse or touchpad, the turn takes place after amovement in a straight line is performed. The software saves a step orseveral steps forward in the stack and then a step with the turn. Thestep of the turn is performed in a single animation to provide a smoothturn. The animation is calculated based on the virtual camera's currentlocation, current direction, the new rotation axis, and the delta in the2D's x-axis of the view, i.e., changes in the left or right direction.

To provide a similar user experience when moving in a backwarddirection, the systems and methods according to the present disclosurerecord the location of the cursor or pointer on the display device afterthe cursor or pointer has been moved by a user via operation of a datainput device such as a mouse, a touchpad, a trackball, or a touchscreen.As discussed herein, the x-coordinates and the y-coordinates of thelocation of the cursor or pointer on the display device is stored in astack and used to determine forward and backward movement of the virtualcamera. When the virtual camera moves backward, the current location anddirection at the current location, along with the previous location anddirection is taken from a stack. During or before backward movement, anaverage direction vector or other smoothing vector optimized forbackward navigation is calculated for two adjacent steps (e.g., steps iand i−1) in the stack. In some embodiments, the smoothing vector isbased on a spline or a Lanczos algorithm being applied to the firstvector and the second vector. In the backward movement, the virtualcamera view corresponding to the calculated average direction vector orother smoothing vector is used. This allows for smoother animation whenperforming turns while moving backward.

Referring now to FIG. 1, the present disclosure is generally directed toa pathway planning system 10 and method for planning a pathway throughan anatomical luminal network of a patient for use during an operation.The pathway planning system 10 may include a computing device 100 suchas, for example, a laptop, desktop, tablet, or other similar device,having a display 102, memory 104, one or more processors 106, and/orother components of the type typically found in a computing device.Display 102 may be touch sensitive and/or voice activated, enablingdisplay 102 to serve as both an input and output device. Alternatively,a keyboard 113, mouse 114, or other data input device may be employed.

Memory 104 includes any non-transitory, computer-readable storage mediafor storing data and/or software that is executable by processor 106 andwhich controls the operation of the computing device 100. In anembodiment, the memory 104 may include one or more solid-state storagedevices such as flash memory chips. In an alternative embodiment, thememory 104 may be mass storage device connected to the processor 106through a mass storage controller (not shown) and a communications bus(not shown).

Although the description of computer-readable media contained hereinrefers to solid-state storage, it should be appreciated by those skilledin the art that computer-readable storage media can be any availablemedia that can be accessed by the processor 106. That is,computer-readable storage media includes non-transitory, volatile andnon-volatile, removable and non-removable media implemented in anymethod or technology for storage of information, such ascomputer-readable instructions, data structures, program modules orother data. For example, computer-readable storage media includes RAM,ROM, EPROM, EEPROM, flash memory or other solid state memory technology,CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetictape, magnetic disk storage or other magnetic storage devices, or anyother medium which can be used to store the desired information andwhich can be accessed by the computing device 100.

Computing device 100 may also include a network module 108 connected toa distributed network or the internet via a wired or wireless connectionfor the transmission and reception of data to and from other sources.For example, computing device 100 may receive computed tomographic (CT)images of a patient from a server, for example, a hospital server, aninternet server, or other similar servers, for use during pathwayplanning. Patient CT images may also be provided to computing device 100via a memory 104, which may be a removable memory.

A pathway planning module 200 includes a software program stored inmemory 104 and executed by processor 106 of the computing device 100. Aswill be described in more detail below, pathway planning module 200guides a clinician through a series of steps to develop a pathway planfor later use during a medical procedure. Pathway planning module 200communicates with user interface module 202 for displaying visualinteractive features to a clinician on the display 102 and for receivingclinician input.

As used herein, the term “clinician” refers to any medical professional(i.e., doctor, surgeon, nurse, or the like) or other user of the pathwayplanning system 10 involved in planning, performing, monitoring, and/orsupervising a medical procedure involving the use of the embodiments ofthe systems, methods, and apparatuses described herein.

Referring now to FIG. 2, in an embodiment, pathway planning using thepathway planning module 200 may be performed in four separate phases. Ina first phase S1, a clinician selects a patient for pathway planning. Ina second phase S2, the clinician adds a target. In a third phase S3, theclinician creates the pathway to the target. Finally, in the fourthphase S4, the clinician reviews and accepts the plan and may export theplan for use in a medical procedure. The clinician may repeat either orboth of the second and third phases S2 and S3 as needed to selectadditional targets and/or create additional pathways for a particularpatient. For example, the clinician may select additional targets andmay create a pathway to each target. The clinician may also oralternatively create multiple pathways to the same target. Once apathway plan is generated, a virtual bronchoscopy view (illustrated inFIGS. 6A-6C) may be displayed, which allows a clinician to navigatewithin the virtual airways of the patient based on the pathway plan.

FIG. 3 is a flowchart illustrating a method 300 for dynamicallydisplaying virtual bronchoscopy views based on position and directioninformation of a virtual camera in accordance with embodiments of thepresent disclosure. At steps 305-310, a location pointed to by datainput device 112 on display 102 within the virtual bronchoscopy view isdisplayed. In some embodiments, the location is set in a known airwaysuch as the trachea or an entry point of the patient. The locationpointed to by data input device 112 on display 102 is obtained as 2Dx-coordinates and y-coordinates. The virtual location is utilized inorder to generate a virtual camera view at the virtual location, at step315. As the clinician progresses through the airways of the patient inthe virtual bronchoscopy view, the clinician clicks or selects variouslocations within the airways to move forwards and backwards.

As the virtual location pointed to by data input device 112 (such as byusing a pointer or cursor) on display 102 arrives at location L_(i) atstep 310 (as further described below with respect to FIG. 7) the viewdirection D_(i) is utilized. Based on the location L_(i), one or moreprocessors 106 determine a view direction D_(i) which corresponds to thelocation L_(i). Location L_(i) is a 2D coordinate having an x-coordinateand a y-coordinate, with respect to a position on display 102 of theairways.

View direction D_(i) may be expressed as a vector having a magnitude andan angle Θ_(i). Angle Θ_(i) may be defined as the difference between theangle of the current vector {tilde over (v)}_(i) and the angle of theprevious vector {tilde over (v)}_(i−1) as illustrated in FIG. 7. Forexample, for a first vector {tilde over (v)}₁ extending from a firstlocation L₁ to a second location L₂ and a second vector {tilde over(v)}₂ extending from the second location L₂ in a direction orthogonal tothe first vector {tilde over (v)}_(i), angle Θ₂ of the second vector{tilde over (v)}₂ is 90°.

Once the location L_(i) and view direction D_(i) are determined, the oneor more processors 106 obtain a virtual camera view C_(i) from memory104 based on the location L_(i) and view direction D_(i), at step 315.Virtual camera view C_(i) is a 2D virtual image to be displayed in thevirtual bronchoscopy window 600 (shown in FIGS. 6A, 6B, and 6C). Virtualcamera view C_(i) is a view inside the airways from the perspective ofthe tip of a virtual camera. Examples of the virtual camera view C_(i)are shown in FIGS. 6A-6C.

At step 320, one or more processors 106 store the first location L_(i)and view direction D_(i) in memory 104, for example, within a stack or alookup table at step S_(i). An example of a lookup table (LUT) is shownbelow.

TABLE 1 Step S_(i) Location L_(i) View Direction D_(i) i − 1 x_(i−1),y_(i−1) Θ_(i−1) i x_(i), y_(i) Θ_(i) i + 1 x_(i+1), y_(i+1) Θ_(i+1) i +2 x_(i+2), y_(i+2) Θ_(i+2) i + 3 x_(i+3), y_(i+3) Θ_(i+3) . . . . . . .. . N x_(n), y_(n) Θ_(n)

As used herein, the term location refers to the coordinate values anddata which indicates the location of the pointer or cursor on display102. View direction refers to the angle difference between the view in astraight line from the current location (a first vector) and the view ina straight line from the previous location (a second vector), as furtherillustrated in FIG. 7.

At step 325, virtual camera view C_(i) is displayed on display 102 inthe virtual bronchoscopy window 600, as shown in FIGS. 6A-6C. In someembodiments, a user may, by clicking a button on mouse 114 or depressinga key or a combination of keys on keyboard 113, alter and update thevirtual camera view C_(i). For example, the user may change thedirection thereby displaying a new virtual camera view at a differentview direction.

Forward Movement

Steps 330-355 of FIG. 3 illustrate movement of a virtual camera in aforward direction through the airways. At step 330, the next locationL_(i+1) and the next view direction D_(i+1) of the virtual camera areobtained based on the location of the pointer or cursor on display 102.The next view direction D_(i+1) may be determined by determining thedirection of the vector extending from location L_(i) to the nextlocation L_(i+1). At step 335, a determination is made as to whether thevirtual camera has moved from location L_(i) to next location L_(i+1) ina forward direction, as described below with reference to FIG. 4.

If it is determined that the virtual camera is moving in a forwarddirection from location L_(i) to next location L_(i+1), the methodproceeds to step 340. At step 340, once both the next location L_(i+1)and view direction D_(i+1) are determined, the one or more processors106 obtains a next virtual camera view C_(i+1) from memory 104 based onthe next location L_(i+1) and next view direction D_(i+1).

At step 345, one or more processors 106 store the next location L_(i+1)and next view direction D_(i+1) in memory 104 at the next step S_(i+1).At step 350, the next virtual camera view C_(i+1) is displayed ondisplay 102 in the virtual bronchoscopy window 600. At step 355, one ormore processors 106 set the current location L_(i+1) at next stepS_(i+1) to current step S_(i), prior to returning to step 330 todetermine next location L_(i+1).

Backward Movement

In some embodiments, if, at step 335, it is determined that the virtualcamera is not moving in a forward direction, the method proceeds to step360. At step 360, it is determined whether the virtual camera is movingin a backward direction. Step 360 is used to confirm that the virtualcamera is moving in a backward direction. Movement in a backwarddirection may be defined as movement to or near a previously-visitedposition. If, at step 360, it is determined that the movement is not ina backward direction, the method returns back to step 335 to determinewhether movement of the virtual camera is in a forward direction.

If, at step 360, it is determined that movement is in a backwarddirection, the method proceeds to step 370. Based on the next locationL_(i+1), which is in a backward direction, the one or more processors106 access a lookup table in memory 104 to determine the previous storedsteps in the lookup table which corresponds to next location L_(i+1).For example, if, at next location L_(i+1), the coordinates are theprocessor would access the lookup table illustrated in Table 1 above anddetermine that previous locations L_(i) and L_(i−1) correspond to stepsS_(i) and S_(i−1), respectively. Once steps S_(i) and S_(i−1) aredetermined, one or more processors 106, at step 370, obtain the viewdirection D_(i) at step S_(i) and the view direction D_(i−1) at stepS_(i−1) from Table 1. Thus, for example, if it is determined thatmovement of the virtual camera is in a backward direction and thelocation corresponds to step S₅, one or more processors 106 obtain theview direction D₄ and the view direction D₃ from the lookup tableillustrated in Table 1.

At step 375, one or more processors 106 calculate a smoothing vector Vbased on view direction D_(i−1) and view direction D_(i). As shown inTable 1 above, view directions D_(i−1) and D_(i) correspond to vectorangles Θ_(i−1) and Θ_(i), respectively. In some embodiments, thesmoothing vector is a new vector having an angle ΘN_(i+1), which bisectsvector angles Θ_(i−1) and Θ_(i). Thus, for backward motion, by applyingsmoothing vector to angle Θ_(i) of view direction D_(i), a new viewdirection at location L_(i+1) is created with a new angle equal toΘ_(i)/2.

In embodiments, the smoothing vector V is a vector that is based on (1)a first or previous vector defined by location L_(i−1) and directionD_(i−1), and (2) a second or current vector defined by location L_(i)and direction D_(i), and optimized for backward navigation so that, forexample, the camera view does not leave the airway and/or the animationof the camera view appears smooth. The smoothing vector V may be avector between the first and second vectors such as a vector that is theaverage or weighted average of the first and second vectors. Thesmoothing vector V may alternatively be determined by using a smoothingalgorithm such as a spline or a Lanczos algorithm. The spline may be aquadratic spline (a spline of degree two) or a cubic spline (a spline ofdegree four).

After determining a new view direction DN_(i+1) at location L_(i+1)based on the smoothing vector, the one or more processors 106 determine,at step 380, a new virtual camera view CN_(i+1) at location L_(i+1). Insome embodiments, location L_(i+1) may be location L_(i−1) or a locationnear location L_(i−1). Thus, instead of using the original virtualcamera view C_(i+1) at location L_(i+1), one or more processors 106obtain the original virtual camera view C_(i+1) from memory 104 andalter virtual camera view C_(i+1) by the new view direction D_(i+1)using the smoothing vector and generates new virtual camera viewCN_(i+1). At step 385, new virtual camera view CN_(i+1) is displayed ondisplay 102 in the virtual bronchoscopy window. Following step 385, oneor more processors 106 set current location L_(i+1) for step S_(i+1) tostep S_(i), at step 390, prior to returning to step 330 to determinenext location L_(i+1).

Turning now to FIG. 4, a flowchart illustrating a method 400 for thedetermination of forward and backward motion of steps 335 and 360 ofFIG. 3 is described in greater detail. Using the current location L_(i)and previous location L_(i−1) of the pointer or cursor on display 102, adetermination can be made of which directions, forwards or backwards,the virtual camera is moving at the current location L_(i).

At step 405, one or more processors 106 obtain the 2D x-coordinates andy-coordinates of current location L_(i) of the pointer on display 102.Next, at step 410, one or more processors 106 obtain the 2Dx-coordinates and y-coordinates of next virtual location L_(i+1) (whichis moved from location L_(i) by a user's operation of data input device112) on display 102.

At step 415, one or more processors 106 determine whether the differencebetween the y-coordinate value of next location L_(i+1) and they-coordinate value of location L_(i) of the pointer on display 102 isless than zero. If, at step 415, it is determined that the differencebetween the y-coordinate value of next location L_(i+1) and they-coordinate value of location L_(i) is less than zero, then one or moreprocessors 106 determine that the virtual camera is moving in a forwarddirection.

If, at step 415, it is determined that the difference between they-coordinate value of location L_(i+1) and the y-coordinate value oflocation L_(i) of the pointer on display 102 is not less than zero,method 400 proceeds to step 420 where it is determined whether thedifference between the y-coordinate value of next location L_(i+1) andthe y-coordinate value of location L_(i) is greater than zero. If, atstep 415, it is determined that the difference between the y-coordinatevalue of next location L_(i+1) and the y-coordinate value of locationL_(i) is greater than zero, then one or more processors 106 determinethat the virtual camera is moving in a backward direction.

Referring now to FIGS. 5 and 6A-6C, 3D map window 500 for a virtualbronchoscopy is displayed. When a target is identified and a pathway isidentified by the computing device 100, a clinician may want to reviewthe pathway in a navigation review mode. FIG. 5 illustrates thenavigation review mode of the planning phase, in which computing device100 shows a 3D map window 500 and a virtual bronchoscopy window 600(FIG. 6A-6C) on the screen of display 102 in accordance with embodimentsof the present disclosure.

The 3D map window 500 shows the 3D map and the virtual bronchoscopywindow 600 shows virtual bronchoscopic video images. The 3D map window500 displays and overlays a pathway 505 to a target 550 and a currentposition indicator 507. In the navigation review mode, the display 102shows the virtual bronchoscopy window 600 as a fly-through view from thetrachea to the target 550.

The virtual bronchoscopy window 600 also shows a pathway 660 toward thetarget 550 for review. The current position indicator 507 moves in the3D map window 500 based on and in accordance with the current positionshown in the virtual bronchoscopy window 600. In an aspect, the pathway660 or 505 may not be displayed based on a display option that aclinician may set between showing the pathway and not showing thepathway.

The virtual bronchoscopy window 600 includes a slider 670 for opacity.By moving the slider 670, opacity of the virtual bronchoscopic videoimages may be changing from opaque to transparent. However, an opacitystatus of the virtual bronchoscopy is not synchronized with the 3D mapshown in the 3D map window 500.

As shown in FIG. 5, airway passage 501 contains three camera views atthree respective locations 510, 520, and 530. As the virtual cameranavigates to target 550 at each of locations 510, 520, and 530, viewdirections 510 a, 520 a, and 530 a, respectively, are displayed withinvirtual bronchoscopy window 600. Each view direction 510 a, 520 a, and530 a corresponds to the virtual bronchoscopic video images shown inFIGS. 6A, 6B, and 6C, respectively. As the virtual camera progressesfrom location 510 to 530, the view seen by a user is altered as shown inFIGS. 6A to 6C, respectively.

In FIG. 6A, a user is able to see, in virtual bronchoscopy window 600,pathway 660 containing locations 510, 520, and 530 along with theforking branches of airway passage 501. As the virtual camera approacheslocation 520 (FIG. 6B), the view shown in virtual bronchoscopy window600 is approaching the forking branches. Once the virtual camera hasreached location 530 (FIG. 6C), an airway passage is displayed atlocation 530. During forward motion, a user moves the pointer or cursorwith an input device such as a mouse, keyboard, or touchpad, and selectslocations forward along pathway 660, such as locations 510, 520, and530. Thus, as a user notices forking branches, a user is able to selectlocations along pathway 660 which enter the forking branches, such aslocation 530. As each location is selected during forward motion, thevirtual camera centers the view at that location. During backwardmotion, as the user moves the pointer or cursor and selects locationsbackward along pathway 660, the systems and methods of the presentdisclosure alter and smooth the view at the backward location, asopposed to displaying a virtual camera view centered at the backwardlocation, thereby providing the user with a camera view which preventsthe view the user sees in virtual bronchoscopy window 600 from beingoutside of the airway. For example, the systems and methods of thepresent disclosure will, for a user selecting backwards motion in theairway of FIG. 6C, prevent the view the user sees from being displayedas if the virtual camera was located outside of the airway at theforking branch of the airways.

FIG. 7 is a graphical illustration 700 of the airways and vectorsassociated with movement within the airways of the Virtual Bronchoscopyview. Graphical illustration 700 shows the vectors and smoothing vectorsat various locations within the airways, such as locations 510, 520, and530 of FIG. 6. As shown in FIG. 7, airway passage 501 contains fivelocations L₀-L₄ (705-740) and respective view directions D₀-D₄ (705a-740 a). Each view direction 705 a-740 a is illustrated as a solidvector line extending from locations 705-740.

In the case of forward motion, as the virtual camera progresses fromlocation L₀ to location L₄, the virtual camera view as seen by a user ateach location 705-740 is displayed in a direction along view directions705 a-740 a, respectively.

In the case of backward motion, for example, traversing locations740-730-720, each view direction displayed at a previous location, e.g.,730 a and 720 a, is altered by smoothing vector. For example, for asmoothing vector that is the average of the current vector and aprevious vector at the current location, altered view direction 730 c isillustrated by a dotted line and is shown as a bisector of 730 a and 730b. Altered view direction 730 c is created as a bisector of the normalview direction 730 a of location L₃ 730, and the view direction ofprevious location L₂ 720, which is illustrated by dashed line 730 bextending from location L₃ 730.

Similarly, for backward motion from location L₃ 730 to location L₂ 720,altered view direction 720 c is illustrated by a dotted line and isshown as a bisector of normal view direction 720 a of location L₂ 720,and the view direction of previous location L₁ 710, which is shown as adashed line 720 b extending from location L₂ 720.

Although the present disclosure has been described in terms of specificillustrative embodiments, it will be readily apparent to those skilledin this art that various modifications, rearrangements and substitutionsmay be made without departing from the spirit of the present disclosure.For example, the determination of the smoothing vector is illustratedabove as being performed online or dynamically during movement of thevirtual camera. It is contemplated that in other embodiments smoothingvectors may be determined offline or prior to the start of thenavigation mode at predetermined locations throughout the airways. Thisother embodiment may be beneficial when utilizing more complex methodsfor determining smoothing vectors or other vectors for optimizing thevirtual camera view when moving in a backward direction. The scope ofthe present disclosure is defined by the claims appended hereto.

In addition to the aforementioned, reference is made to followingcommonly assigned applications, which describe features of imageprocessing and user-interface updating, among other features which arerelevant to the systems described herein: U.S. Provisional PatentApplication No. 62/020,240, entitled “System And Method For NavigatingWithin The Lung,” filed on Jul. 2, 2014; U.S. Provisional PatentApplication No. 62/020,220 entitled “Real-Time Automatic RegistrationFeedback,” filed on Jul. 2, 2014; U.S. Provisional Patent ApplicationNo. 62/020,177, entitled “Methods for Marking Biopsy Location,” filed onJul. 2, 2014; U.S. Provisional Patent Application No. 62/020,242,entitled “Unified Coordinate System For Multiple CT Scans Of PatientLungs,” filed on Jul. 2, 2014; U.S. Provisional Patent Application No.62/020,245, entitled “Alignment CT,” filed on Jul. 2, 2014, by Klein etal.; U.S. Provisional Patent Application No. 62/020,250, entitled“Algorithm for Fluoroscopic Pose Estimation,” filed on Jul. 2, 2014;U.S. Provisional Patent Application No. 62/020,253, entitled “TracheaMarking,” filed on Jul. 2, 2014; U.S. Provisional Patent Application No.62/020,261, entitled “Lung And Pleura Segmentation,” filed on Jul. 2,2014; U.S. Provisional Patent Application No. 62/020,258, entitled “ConeView—A Method Of Providing Distance And Orientation Feedback WhileNavigating In 3D,” filed on Jul. 2, 2014; U.S. Provisional PatentApplication No. 62/020,262, entitled “Dynamic 3D Lung Map View for ToolNavigation Inside the Lung,” filed on Jul. 2, 2014; U.S. ProvisionalPatent Application No. 62/020,261 entitled “System and Method forSegmentation of Lung,” filed on Jul. 2, 2014; and U.S. ProvisionalPatent Application No. 62/020,257, entitled “Automatic Detection OfHuman Lung Trachea,” filed on Jul. 2, 2014. All these references aredirected to aspects of processing the DICOM images, detecting thetrachea, navigating within the lung, and displaying the DICOM images andprocessed images to provide enhanced clarity and performance foranalysis, diagnosis, and treatment systems relating to, among otherthings, lung treatment planning and navigation. The contents of all theabove-referenced applications are incorporated herein by reference.

What is claimed is:
 1. A method for displaying virtual bronchoscopyviews while navigating through an airway of a virtual bronchoscopy, themethod comprising: determining a location corresponding to movementthrough the virtual bronchoscopy; determining whether the movement is ina forward direction or a backward direction; in response to determiningthat the movement is in the forward direction, displaying a forwardvirtual camera view at the determined location; and in response todetermining that the movement is in the backward direction: retrieving afirst direction and a second direction from memory; determining asmoothing vector at the determined location based on the first directionand the second direction; obtaining a smoothed virtual camera view basedon the smoothing vector; and displaying the smoothed virtual camera viewat the determined location.
 2. The method according to claim 1, whereindetermining whether the movement is in a forward direction or a backwarddirection includes: determining a next pointer location and a currentpointer location on a virtual bronchoscopy screen; calculating adifference between a coordinate of the next pointer location and acoordinate of the current pointer location; in response to thecalculated difference being positive, determining that the movement isin the backward direction; and in response to the calculated differencebeing negative, determining that the movement is in the forwarddirection.
 3. The method according to claim 1, wherein the location isdetermined based on a location of a pointer or cursor on a screen whichdisplays virtual camera views.
 4. The method according to claim 1,wherein determining the smoothing vector includes: determining a firstvector based on the first direction; determining a second vector basedon the second direction; and averaging the first vector and the secondvector to obtain the smoothing vector.
 5. The method according to claim1, wherein determining the smoothing vector includes: determining afirst vector based on the first direction; determining a second vectorbased on the second direction; and applying a spline or a Lanczosalgorithm to the first vector and the second vector to obtain thesmoothing vector.
 6. The method according to claim 5, wherein the splineis a spline of order two or a spline of order four.
 7. The methodaccording to claim 1, further comprising: receiving an input from a userwhich alters the forward virtual camera view or the smoothed virtualcamera view; and storing the altered, forward virtual camera view or thealtered, smoothed virtual camera view in the memory.
 8. The methodaccording to claim 1, wherein determining the smoothing vector includesdetermining a vector having a direction between the first direction andthe second direction.
 9. An apparatus for displaying virtualbronchoscopy views while navigating through an airway of a virtualbronchoscopy, comprising: a memory storing instructions, a plurality ofvirtual camera views of the virtual bronchoscopy, a first direction, anda second direction; a processor configured to execute the instructions,wherein the instructions, when executed by the processor, cause theprocessor to: determine whether movement through the airway of thevirtual bronchoscopy is in a backward direction; in response todetermining that the movement is in the backward direction: retrieve thefirst direction and the second direction from the memory; and determinea smoothing vector at a current location based on the first directionand the second direction; and a display configured to display, on ascreen, an image of a smoothed virtual camera view corresponding to thedetermined smoothing vector at the current location.
 10. The apparatusaccording to claim 9, wherein the current location is determined basedon a location of a pointer on the screen.
 11. The apparatus according toclaim 9, wherein the instructions, when executed by the processor,further cause the processor to: determine a first pointer location and asecond pointer location on the screen displaying a virtual camera view;calculate a difference between a coordinate of the first pointerlocation and a coordinate of the second pointer location; and inresponse to the calculated difference being positive, determine that themovement is in the backward direction.
 12. The apparatus according toclaim 9, wherein the instructions, when executed by the processor,further cause the processor to: determine a first vector based on thefirst direction; determine a second vector based on the seconddirection; and average the first vector and the second vector to obtainthe smoothing vector.
 13. The apparatus according to claim 9, whereinthe instructions, when executed by the processor, cause the processor todetermine the smoothing vector having a direction that is between thefirst direction and the second direction.
 14. The apparatus according toclaim 9, wherein the instructions, when executed by the processor,further cause the processor to: determine a first vector based on afirst direction; determine a second vector based on a second direction;and apply a spline to the first vector and the second vector to obtainthe smoothing vector.
 15. The apparatus according to claim 14, whereinthe spline is a spline of order two or a spline of order four.
 16. Amethod for displaying virtual bronchoscopy views of an airway, themethod comprising: determining a location corresponding to movementthrough the airway of a virtual bronchoscopy; determining whether themovement is in a forward direction or a backward direction; in responseto determining that the movement is in the forward direction, displayinga forward virtual camera view at the determined location; and inresponse to determining that the movement is in the backward direction:obtaining a smoothing vector; obtaining a smoothed virtual camera viewat the determined location based on the obtained smoothing vector; anddisplaying the smoothed virtual camera view at the determined location.17. The method according to claim 16, further comprising: determining afirst vector based on a first direction; determining a second vectorbased on a second direction; and averaging the first vector and thesecond vector to obtain the smoothing vector.
 18. The method accordingto claim 16, further comprising: determining a first vector based on afirst direction; determining a second vector based on a seconddirection; and applying a spline or a Lanczos algorithm to the firstvector and the second vector to obtain the smoothing vector.
 19. Themethod according to claim 18, wherein the spline is a spline of ordertwo or a spline of order four.